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Northwestern University and University of Pennsylvania 

A test is said to control for type I error if it is unlikely to re- 
ject the data-generating process. However, if it is possible to produce 
stochastic processes at random such that, for all possible future re- 
alizations of the data, the selected process is unlikely to be rejected, 
then the test is said to be manipulable. So, a manipulable test has 
essentially no capacity to reject a strategic expert. 

Many tests proposed in the existing literature, including calibra- 
tion tests, control for type I error but are manipulable. We construct 
a test that controls for type I error and is nonmanipulable. 

1. Introduction. Professional forecasts are often presented as probabilis- 
tic statements [see, e.g., Gneiting and Raftery (2005) for a review of the role 
of probabilistic forecasts in meteorology]. The quality of these forecasts is 
regularly tested empirically. A concern [which can be traced back at least 
to Brier (1950)] is that if forecasts are tested, then experts may misreport 
their forecasts with the intention of passing the test. Recent literature shows 
that, without any knowledge about the data-generating process, it is possi- 
ble to produce forecasts that pass some empirical tests on all possible future 
realizations of the data. 

An example of a test that can be manipulated in this way is the well- 
known calibration test. Suppose that a stochastic process generates in every 
period an outcome that can be either or 1. The calibration test requires 
the empirical frequency of 1 to be close to p in the periods that 1 was fore- 
casted with probability close to p. Dawid (1982) shows that the forecasts of 
the data-generating process will eventually be calibrated. Foster and Vohra 
(1998) show that any individual can produce forecasts with a random device 
such that, for all possible infinite strings of zeros and ones, the realized fore- 
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casts will eventually be calibrated as well, with probability one according to 
the random device used by the individual. 

It is natural to seek tests that cannot be manipulated. We address this 
problem in the present paper. We study the following framework: A stochas- 
tic process generates in every period an outcome that can be or 1 (we make 
no assumptions, such as an independent, identically distributed process). Be- 
fore any data are observed, an expert named Bob delivers a theory, defined 
as a mapping that takes as an input any finite string of outcomes and re- 
turns as an output a probability of 1; equivalently, theories are probability 
measures P on the space of infinite sequences of outcomes. 

A tester named Alice tests Bob's theory P by selecting an event Ap (i.e., 
a set of sequences of outcomes), which Alice regards as consistent with the 
theory P. We call Ap the acceptance set and its complement the rejection 
set for the theory P. If, for every P, the event Ap has high probability 
according to P, then the data-generating process will not be rejected (with 
high probability). We then say that the test controls for type I error (of 
rejecting the data-generating process). 

Assume that Bob knows nothing about the data-generating process. How- 
ever, Bob may use a random device C to select his theory P. At first, Alice 
cannot tell whether the announced theory coincides with the data-generating 
process or was selected at random. This must be determined by the data. 
If, for any sequence of outcomes, Bob's theory P is not rejected with high 
probability according to Bob's random device ^, then we say that the test 
can be manipulated (with this high probability), or we say, equivalently, that 
the test has essentially no capacity to reject theories produced strategically. 

Many tests that control for type I error have essentially no capacity to 
reject theories produced strategically. The calibration test can be manip- 
ulated. Several extensions of the calibration test have also been proven to 
be manipulable [see, e.g., Fudenberg and Levine (1999), Lehrer (2001) and 
Sandroni, Smorodinsky and Vohra (2003)]. Other statistical tests can also 
be manipulated. A prequential test rejects or accepts a theory based only 
on the observed data sequence and the next period forecasts made by the 
theory along the realized sequence of outcomes [see Dawid (1991) for more 
details on the prequential principle]. Many standard statistical tests, in- 
cluding calibration tests, are prequential tests, and prequential tests can be 
manipulated [see Sandroni (2003), Vovk and Shafer (2005), Olszewski and 
Sandroni (2008) and Shmaya (2008)]. 

Dekel and Feinberg (2006) show that, under the continuum hypothesis, 
there exists a test that does not reject the data-generating process and 
cannot be manipulated because every random device ( fails this test with 
certainty on an uncountable number of paths. We construct a (nonprequen- 
tial) test, called the global category test, that also does not reject the data- 
generating process with probability one and cannot be manipulated. How- 
ever, our construction does not assume the continuum hypothesis; that is, it 
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is performed within the Zermelo-Fraenkel axioms and the axiom of choice. 
In addition, the global category test is explicitly constructed; that is, we 
give set-theoretic formulas for the acceptance sets. 

The significance of dispensing the continuum hypothesis can be seen in 
a simple example. Assume that Bob announces an easy to describe theory 
(e.g., and 1 occur with probability 0.5 in all periods) and the sequence of 
outcomes follows an easy-to-describe deterministic process (e.g., in every 
third period and 1 otherwise). If Alice uses the test in Dekel and Feinberg 
(2006), no researcher can determine whether Alice rejects Bob's theory. If 
she uses the global category test, this is a straightforward determination. 

The global category test cannot be manipulated, in the sense that every 
random device C fails this test with certainty on all paths except a first 
category set of them. 

The paper is organized as follows. In Section 2, we introduce our basic 
concepts and show some classic examples of tests (calibration and likelihood 
tests). In Section 3, we construct the global category test and provide an 
informal discussion of nonmanipulability. In Section 4, we show that the 
global category test can be modified to address the problem that, in practice, 
testers observe only finite data sets. We also show that the global category 
test can be modified so as to belong to the familiar class of likelihood tests. In 
addition, we exhibit a large class of tests that are not necessarily prequential 
tests, but can be manipulated. Finally, still in Section 4, we offer some 
additional results on the manipulability of random prequential tests and an 
informal discussion on the implications (for the prequential principle) of the 
finding that prequential tests can be manipulated while some nonprequential 
tests cannot be manipulated. Proofs are relegated to the Appendix. 

2. Basic concepts. Each period one outcome, or 1, is observed (our 
results generalize to any finite number of outcomes per period). Let = 
{0, 1}°° be the set of all paths, that is, infinite histories. A finite history 
Sm S {0, 1}™, m >t, (or a path s G ri) is an extension of a history sj G 
{0, 1}* if the first t outcomes of Sm or s coincide with the outcomes of st- In 
the opposite direction, let Sm \ t {ov s \ t) be the history st € {0, 1}* whose 
outcomes coincide with the first t outcomes of or s. A cylinder with base 
on st is the set C(sj) C {0, 1}°° of all infinite extensions of st- We endow 

with the topology that compares unions of cylinders with a finite base. 
Let be the algebra that consists of all finite unions of cylinders with base 
on {0, 1}*. Denote by N the set of natural numbers. Let 9 be the a-algebra 
generated by the algebra 9° := UteAf ^t] that is, 9 is the smallest cr-algebra 
that contains Q^. 

Let A(r2) be the set of all probability measures on (0, ?>). We endow A{0,) 
with the weak*-topology and the c-algebra of Borel sets (i.e., the smallest 
c7-algebra that contains all open sets in weak*-topology). 
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As is well known, the weak*-topoIogy consists of all unions of finite inter- 
sections of sets of the form 

{Q eA{n):\E^h- E^h\<e}, 

where E stands for the expected- value operator P € A(rj), e > 0, and h is 
a real- valued and continuous function on $7. We refer the reader to Rudin 
(1973) for additional details on the weak*-topology. We also define AA(il) 
as the set of probability measures on A(r2). 

Before any data are observed, an expert named Bob announces a prob- 
ability measure P G A(r2), which (Bob claims) describes how nature will 
generate the data. To simplify the language, we call a probability measure 
a theory. A tester named Alice tests Bob's theory empirically. 

Definition 1. A test is a function T : x A{fl) {0, 1}. 

That is, a test is defined as an arbitrary function that takes as input a 
theory and a path and returns a verdict that is or 1 . When the test returns 
a 1, it does not reject (or, simply, accepts) the theory. When a is returned, 
the theory is rejected. 

Any test divides paths into those in Ap := {s € 17 | T(s,P) = 1}, where 
the theory P is accepted, and those in Q — Ap, where the theory is rejected. 
The set is called the acceptance set, and its complement Q — Ap is called 
the rejection set. We consider only tests T such that the acceptance sets 
Ap are measurable. 

Given a theory P € A(il), a path s E and a history st = s \ t, let 

/o^(.):=P(C(l)) and /f (-) - 

be forecasts made along s. ffis) is arbitrarily defined as when P{C{st)) = 
0. The forecasts of P and P' are equivalent along s if ft'is) = ff (s) for all 
periods t>0. A test T is prequential if for any given two theories P and P' , 
equivalent along s, T{s, P) = if and only if T(s, P') = 0. So, a prequential 
test rejects or accepts a theory based only on the forecasts made by the 
theory along the realized path. Fix any e G [0, !]• 

Definition 2. A test T does not reject the data-generating process with 
probability 1 — e if for any P £ A{Q) 

P{Ap) >l-e. 

That is, a test does not reject the data-generating process (with high 
probability) if, no matter which probability measure P generates the data, 
P is not likely to be rejected according to its own probability distribution 
over paths. 
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2.1. Classic examples of tests. Given a path s let It{s) be the tih 
outcome of s. The test 

1 

T{s,P) = 1 if and only if lim - - It{s)] = 

n—*oo n '—^ 

t=l 

requires the average forecast of 1 to match the empirical frequency of 1. 

The test in Lehrer (2001) requires the match between average forecasts 
and empirical frequencies to occur on several subsequences. The calibration 
test requires the empirical frequency of 1 to be near p G [0, 1] in the periods 
in which 1 was forecasted with probability close to p. 

One well-known class of tests not based on matching empirical frequencies 
is likelihood tests. Let Q : A{0,) A{il.) be any function that takes a theory 
P as an input and returns an alternative theory Qp as an output. The 
likelihood test T is defined by 

T{s,P) = l iSyneNP{C{sn))^0 

and 

y Qp{C{Sn)) , I 
1™ sup r— < OO, Sn = s\n. 

n-ico r\yj\ySn)} 

This test requires that the likelihood of Qp does not become arbitrarily 
larger than the likelihood of P. A proof that calibration and likelihood tests 
do not reject the data-generating process with probability one can be found 
in Dawid (1982, 1985). 

Calibration tests are prequential tests because, like many other well- 
known tests, they take as an input not the entire theory but only the data 
and the forecasts made by the theory along the realized sequence of out- 
comes. A likelihood test may or may not be a prequential test, depending 
upon the way the alternative theory is selected as a function of P. In 
Section 4, we exhibit examples of prequential and nonprequential likelihood 
tests. 

2.2. Manipulating tests. Foster and Vohra (1998) show that Bob can 
pass the calibration test on all paths if he is allowed to select theories at 
random. More precisely, the calibration test can be manipulated with prob- 
ability one according to the following definition: 

Definition 3. A test T can be manipulated with probability 1 — e if 
there exists a random generator of theories Ct £ AA(il) such that, for every 
path s G 0, 



CT{{P(^m)\T{s,P) = l])>l-e. 
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Naturally, our definition of manipulability requires a measurability provi- 
sion on the sets {P € A(Q) ■.T{s,P) = 1}. However, to ease the exposition, 
we deal with measurability issues in the Appendix. 

Fudenberg and Levine (1999), Lehrer (2001), Sandroni, Smorodinsky and 
Vohra (2003) show that generalized calibration tests can also be manipu- 
lated. Sandroni (2003), Vovk and Shafer (2005), Olszewski and Sandroni 
(2008) and Shmaya (2008) show that prequential tests that can be manip- 
ulated. A partial review of this literature can be found in Cesa-Bianchi and 
Lugosi (2006). We also refer the reader to Gneiting, Balabdaoui and Raftery 
(2007) for additional comments on this literature. 

The random generator (^t may depend only on the test T. It follows that, 
even if Bob does not know the data-generating process, he can be very 
confident that by selecting theories at random before any data are observed 
he will not fail any of these tests, no matter which data are realized in the 
future. 

The crucial property we seek is the existence, for every random generator 
of theories, of a path at which rejection may occur (because it ensures that 
it is feasible to reject strategically produced theories). However, stronger 
properties may be obtained. Alice may be interested in paths such that 
randomly produced theories fail the test with near certainty (as opposed to 
probability higher than e) and such that the sets of these paths are larger 
than a single path. 

Definition 4. Fix a test T. Given a random generator of theories C G 
AA(ri) and e > 0, let i?^ C be the set of all paths s G O such that 

C{{PeA{n)\T{s,P) = 0})>l-e. 

The set is called the revelation set, where the random generator of 
theories ^ fails to manipulate the test with probability 1 — e. 

In Section 3, we exhibit a test such that for all random generators of the- 
ories the revelation sets are always non-empty (and topologically large). 

3. A nonmanipulable test. In this section, we construct a test that can- 
not be manipulated, that is, a test with nonempty revelation sets. In addi- 
tion, we show that the revelation sets are topologically large in the following 
sense: Given a subset A of a (complete metric) space, let A be the closure 
of A. A set A is called nowhere dense when the interior of its closure A 
is empty. A first- category set is a countable union of nowhere-dense sets. A 
first-category set may be regarded as (topologically) small. The complement 
of a first-category set may be regarded as (topologically) large. We refer the 
reader to Oxtoby (1980) for these definitions and some basic results regard- 
ing first-category sets. 
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We now define our test. Let S = {s^,s^, . . .} be a countable dense subset 
of Q (i.e., for every cylinder C with finite base, there exists a path that 
belongs to C; e.g., C can be the set of paths with all but a finite number 
of outcomes equal to zero). Fix any k £ N. For every path s*, there exists 
a period t G N such that the cylinder C{sl) with base on the finite history 
sl = \t satisfies 

(3.1) P{C{si)-{s'})<^. 

Indeed, the sequence of sets C(sJ) — {s*} is descending (as t goes to infin- 
ity), and its intersection is empty. So, P{C{s\) — {s*}) goes to zero as t goes 
to infinity. Let t{i,k,P) be the smallest natural number such that (3.1) is 
satisfied. 

Given a dense set of paths S = {s^, s^, . . .}, the global category test Ts can 
be defined as follows: 

{oo oo 
0, if s^^andsG fl |JC(4(ifcp)); 
fc=ii=i 
1, otherwise. 

Let be the revelation set for the random generator of theories C ^ 
AA(!^). 

Theorem 1. Fix any countable dense set S C^. The global category 
test Ts does not reject the data- generating process with probability one. Given 
any random generator of theories C G AA(f]), the revelation set is the 
complement of a first- category set of paths. 

The global category test controls the type I error of rejecting the data- 
generating process and it cannot be manipulated. If the data-generating 
process is announced, then it passes the global category test with probability 
one. On the other hand, no matter which random generator of theories ^ is 
employed, failure is inevitable on the paths of the revelation set R^. This 
set is nonempty (and topologically large). 

The main objective of this paper is to construct a test that is nonmanip- 
ulable according to Definition 3, that is, in the sense studied in the existing 
literature. Naturally, our result is limited to our definition of manipulation. 
Alternative definitions of strategic manipulation are beyond the scope of 
this paper. 

The global category test can be combined with any other test to produce 
a harder test. Even so, no test can avoid the following difficulty: for any 
finite collection of probability measures Qi, . . . , Qt over outcome paths, and 
for any test T that does not reject the data-generating process with high 
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probability, Bob can ensure that his randomly selected theory is unlikely to 
be rejected on a set of outcome paths whose Qj -probabilities {j = I, . . . ,k) 
are arbitrarily close to one [see Sandroni and Olszewski (2008)]. 

Since the proof of Theorem 1 is somewhat involved, it is relegated to the 
Appendix, but the key idea of this proof is simple. The rejection sets for the 
global category test have been defined as an intersection of unions of small 
cylinders Ci, C2, . . . around paths s^, s^, . . . (from which the paths s^, s^, . . . 
have additionally been removed). The fact that a cylinder around each path 
is included in the rejection set guarantees that, for any single theory, the 
rejection set is topologically large; moreover, the fact that those cylinders 
are small guarantees that the data-generating process will not be rejected. 

Suppose now that we are given a random generator of theories Since 
cylinders Ci, C2,... are defined for every single theory around the fixed 
set of paths s^, s^,..., it follows that sufficiently small cylinders around 
those paths s^, s^, . . . will be contained in cylinders Ci, C2, . . . , respectively, 
for a set of theories whose (^-probability is sufficiently close to one. As the 
intersection of topologically large sets is itself a topologically large set, there 
exists a topologically large set contained in the rejection sets for a set of 
theories whose (^-probability is equal to one. 

3.1. Informal discussion on manipulability. A result concerning the ma- 
nipulability of a test can be interpreted in different ways. Consider a fore- 
caster whose objective is to be calibrated. The Foster and Vohra (1998) 
result shows that the forecaster's goal can be achieved no matter how the 
data evolves in the future. Hence, from this forecaster's perspective, the Fos- 
ter and Vohra (1998) result is positive. The results of Sandroni (2003), Vovk 
and Shafer (2005), Olszewski and Sandroni (2008) and Shmaya (2008) show 
that, like calibration, many observable properties of the data-generating 
process can be obtained by a strategic forecaster, no matter how the data 
evolves in the future. So, these results show how to produce forecasts that, 
in the future, will prove to have some observable properties of the data- 
generating process. However, these strategically produced forecasts may re- 
main bounded away from the predictions of the data-generating process, 
and these forecasts need not have all observable properties of the data- 
generating process simultaneously (only the properties used to define some 
specific manipulable test). Hence, whether strategically produced forecasts 
are desirable from the perspective of a forecaster is an arguable point, as this 
may depend upon the objective of the forecaster. Now, consider the concept 
of manipulability from the viewpoint of a tester. 

Assume that Alice wants to determine whether Bob has information about 
the data-generating process that she does not have. If this is Alice's objec- 
tive, then a manipulable test (e.g., the calibration test) has limited use to her 
when Bob knows the test that will be used (see Section 4.2 for a discussion 
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of the case in which Bob does not know the test at the time of dehvering his 
theory) . Even in the extreme case that Bob is completely uninformed about 
the data-generating process, he can strategically pass such a test. Hence, 
Alice knows from the outset (i.e., before any data is revealed) the verdict 
that, with near certainty, the test will deliver once the data is revealed. 

The difficulty with manipulable tests can be understood in the context 
of a contracting problem between a tester (Alice) and an expert (Bob) who 
claims, before any data is observed, to know the data-generating process. 
Alice does not know the data-generating process, and she is willing to pay 
a reward that gives Bob utility u> if he announces his theory at period 
zero. In order to discourage Bob from delivering an arbitrary theory, Alice 
stipulates a penalty if Bob's theory is rejected by her test. This penalty 
gives Bob disutility d> u (Bob does not discount the future) . Bob observes 
Alice's test before deciding whether to accept Alice's contract. Bob receives 
no reward and no penalty if he does not accept the contract. 

Alice looks for a screening contract, which will be accepted by Bob if 
informed about the data-generating process and rejected by Bob if unin- 
formed about the data-generating process. Then, Alice learns whether Bob 
is informed from Bob's choice on whether to accept the contract. 

Assume that the test does not reject the data-generating process with 
probability 1 — e, where e is small enough so that u — > and u — d{l — £) < 
0. Bob, if informed, accepts the contract because his expected utility with 
the contract is positive (and without the contract his utility is zero). On 
the other hand, if Bob is uninformed, then he faces uncertainty. He does 
not know the odds that any theory is rejected. Assume that Bob evaluates 
his prospects based on the minimum expected utility he obtains. This is 
the most pessimistic behavioral rule of decision under uncertainty among 
those axiomatized by Gilboa and Schmeidler (1989). Formally, if Bob is 
uninformed and uses a random generator of theories C £ AA(il), then his 
payoff, with a contract, is 

(3.2) u-dsupC{PG A(0) |r(s,P) = 0}. 

s&S 

If Alice's test is manipulable, then Bob, although extremely averse to 
uncertainty, accepts her contact because his payoff with the contract (3.2) is 
positive. Hence, a manipulable tests cannot be used to construct a screening 
contract that Bob, if informed, accepts and if uninformed, does not accept. 

On the other hand, if Alice's test is nonmanipulable, then Bob, when un- 
informed, does not accept Alice's contract, as his payoff with the contract is 
negative. So, a nonmanipulable test does produce a screening contract. Bob 
accepts this contract (and, therefore, Alice pays Bob) only when Bob deliv- 
ers to Alice something she values (the data-generating process), as opposed 
to a theory selected by a method that she can produce on her own without 
having to pay an expert for it. 
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3.2. Limitations of the main result. The main objective of this paper is 
to construct a nonmanipulable test. Nonmanipulabihty is in our opinion a 
desirable property (at least as far as Bob knows in advance the test that will 
be used), because it ensures the feasibility of the rejection of a strategic, but 
otherwise completely uninformed, expert. However, we do not claim that an 
uninformed expert is likely to be rejected by a global category test, nor do 
we claim that a theory that passes our test must be closely related to the 
data-generating process. 

In addition, we are not confidently advocating the use of global category 
tests. An uninformed expert may accept Alice's contract, even if she adopts 
a global category test, when Bob adopts a less pessimistic behavioral rule 
to evaluate uncertain prospects. Moreover, global category tests have not 
been extensively studied, and they may have undesirable properties. On the 
other hand, global category tests do not reject the data-generating process 
with probability one. Hence, they can be combined with any other test 
without reducing the odds that the data-generating process is rejected. This 
combined test remains nonmanipulable. 

4. Additional results. In this section, we show modified forms of the 
global category test and discuss some of its properties. We begin with the 
simple observation that tests, as defined in Section 2, give a verdict only after 
observing an infinite history. In practice, a tester can only observe finite data 
sets and, therefore, may find more applicable tests that give some verdict 
with a finite number of outcomes. 

Definition 5. Rejection tests have the property that, for any theory 
P G A(r2), the rejection set il. — Ap is a union of cylinders. 

So, a rejection test rejects a theory in finite time. 

Definition 6. A test T2 is harder than a test Ti if 

{sGQ: T2{s, P) = l}Q{sen: Ti{s, P) = 1}. 

If T2 is harder than Ti, then rejection by Ti implies rejection by T2. 

Proposition 1. Fix any 6 e {0,1 - e]. Let Ti be any test that does 
not reject the data- generating process with probability 1 — e. There exists 
a rejection test T2 that is harder than the test Ti and does not reject the 
data- generating process with probability 1 — e — 5. 

Proposition 1 is a direct corollary of the following well-known result: for 
any given probability measure P £ A{Q) and 6 > 0, any set ^ G S can be 
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enlarged to an open set U D A such that P{U) < P{A) + 6 [see Ulam's 
theorem, Theorem 7.1.4 in Dudley (1989)]. 

Proposition 1 shows that, given any test Ti, there exists a rejection test 
T2 that does not reject the data-generating process with probability almost 
as high as Ti, and the acceptance sets of T2 are contained in those of Ti. 
Thus, T2 is more capable than Ti to reject theories. For example, if Ti has 
nonempty (or topologically large) revelation sets, then T2 is a rejection test 
with nonempty (or topologically large) revelation sets. We refer to T2 as the 
rejection test associated with the test Ti. 

Given e > 0, let be a rejection test (that does not reject the data- 
generating process with probability 1 — e) associated with a global category 
test Ts- Corollary 1 follows immediately from Proposition 1 and Theorem 1. 

Corollary 1. The rejection test T*" does not reject the data- generating 
process with probability 1 — £, and it cannot be manipulated. The revelation 
sets of T^ contain the revelation sets of the global category test, and, hence, 
their complements are first- category sets. 

For any given theory P, the test T'^ delivers the rejection set of the the- 
ory P that consists of finite histories of outcomes that must be regarded as 
sufficiently inconsistent with the theory P to justify its rejection. This test 
does not reject the actual data-generating process and, in contrast with ma- 
nipulable tests, maintains the possibility of theory rejection (in finite time) 
even if an uninformed individual randomizes with the intent of manipulating 
the test. In addition, it follows directly from our proof that each revelation 
set of T"" contains an open and dense set. Hence, the revelation sets of T^ 
contain finite- histories. 

4.1. A nonmanipulable likelihood test. In this section, we modify a global 
category test so that, in this modified form, it belongs to the familiar class 
of likelihood tests (and remains nonmanipulable). 

Let an atom of a theory P be a path that P assigns strictly positive 
measure. Let A C A(0) be the set of all theories that attach probability one 
to a set comprising finitely many atoms. 

Let Ts be a global category test. Given any natural number m > 1, let 
be a rejection test that is harder than T5 and does not reject the data- 
generating process with probability 2(rra+i)^ • Given a theory P, let EJp be 
the rejection set of P (for test T™). So, by definition. 



^ ^' - 2(m + l)3 
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Given any theory P G {AY, let Cp be a cylinder with finite base such 
that 

(4.1) 0<P{CP)< \ 

The existence of a cylinder that satisfies (4.1) is shown in Remark A.l 
in the Appendix. Let Rf := i?^ U C^, and let be the theory P e {Af 
conditional of i?p ; that is, for every A e Si, 

(4.2) ^?^(^) = ^^S^- 

Let 7r(m) = . Let Qp be the theory defined by 



QP=Y1 7r(m)Q^. 

m=l 

Given a theory P € (AY, Qp is well defined because 

oo 

^("^) = 1- 

m=l 

The function Q:A{n) — > A(Q), such that Q{P) = Qp if P G (^)= and 
Q(P) = P if P G ^, defines the likelihood test f . 

Proposition 2. T/ie likelihood test T is harder than the global category 
test Ts ■ 

The likelihood test T, like any other likelihood test, does not reject the 
data-generating process with probability 1. It follows from Proposition 2 
that the revelation sets of the likelihood test T contain those of the global 
category test T5. Hence, the likelihood test T is nonmanipulable. 

4.2. The prequential principle. Prequential tests are manipulable, but 
some nonprequential tests (e.g., the global category tests) are nonmanipu- 
lable. These results pose a difficulty for the prequential principle, because 
they indicate that the prequential principle must be discarded to produce a 
nonmanipulable test. We examine in closer detail the relationship between 
manipulability and the prequential principle. We now consider random tests. 
That is, we allow Alice to randomize among prequential tests that do not 
reject the data-generating process. 

Let {Q,B,v) be a probability space where is a parameter space, B is 
a cj-algebra and v is a probability measure on {Q,B). A random test is 
probability space {Q,B,v) and a function T:QxQx A{Q) {0,1}. So, 
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for every parameter 9ee,fg:nx A{n) {0,1}, fg{s,P) =f{e,s,P) is 
a test. We also define r,(^,P) = f{e,s,P) and fp{e,s) =f{e,s,P). We 
assume that T is measurable, jointly, with respect to 6, s and P (and we 
refer to this assumption as the joint measurability condition). 

A random test is prequential if, for every 6 £@, Tq is a prequential test. 
A random test does not reject the data-generating process with probability 
one if, for every P € A(0), 

{v X P){{{9, s)eexn: fp{9, s) = 1}) = 1. 

We now construct a prequential random test T that does not reject the 
data-generating process with probability one. Given a countable dense set 
S = {s^, s^, . . .}, let € A{Q) be a probability distribution defined by 



i{i + 1) 

for i = 1, 2, . . . and Q^{Q — S) = 0. That is, Q'^ assigns full measure to S. 
Since 

oo 2 

is well defined. 

Let Tg^ be the likelihood test such that, for every theory P, the alterna- 
tive theory is . This test is prequential because the alternative theory is 
fixed independently of the theory announced by Bob. By general properties 
of likelihood ratio tests, T^^ does not reject the data-generating process 
with probability 1 [see Dawid (1982)]. 

Let = ri = {0, 1}°° be the parameter space. Given a path 9 € Q, let 
iSg C r2 be the set of all paths that coincide with 9 in all but a finite number 
of periods. That is, 

Sg = {.sen:\{t:It{s)j^It{9)}\<oo} 

oo 

= U{O,l}*"^x{/,(^)}x{/,+i(0)}x...; 
t=i 

of course the set Sq is countable and dense in 0,. We define the randomized 
likelihood test T^^ as follows. First, let A[Q) C A{Q,) be the set of all theories 
that assigns zero measure to any single path. Alice draws randomly a path 
9 according to a probability measure v € A{Q) and then tests Bob with 
the likelihood test TJ;^. So, f^^{9,s,P) = T^^{s,P). 

To make this definition precise, we must say how we represent the set Sg 
as a sequence Sg, Sg, . . . , because the probability distribution Q"^* depends 
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on the order of the paths m Sg. In addition, if we took an arbitrary rep- 
resentation, the test T^^ could potentially violate our joint measurability 
condition. 

One possible way to represent the set Sg as a sequence of paths si, Sg, ... is 
such that for every t = 1,2, . . . the paths from {0, 1}* x x {It+i{9)} x 

• • • precede the paths from {0, 1}* x {lt+i{0)} x {It+2{d)} x {0, 1}*"^ x 

{It{e)} X {It+i{9)} X and paths from {0,1}* x {/t+i(0)} x {It+2{9)} x 

• • • — {0, 1}*"^ x {lt{0)} X {It+i{9)} X • • • are ordered lexicographically. As 
shown in the Appendix (Lemma A. 2), the joint measurability condition is 
satisfied with the sets Sg ordered in this way. 

Bob knows that he is tested according to this protocol, but he does not 
know the test selected by Alice. Given any random generator of theories 
C G AA(Q), let R'^{e) be the revelation set of the test T^^^. 

Proposition 3. Consider a randomized likelihood test T^^, {^,Q,v), 
V G A(r2). For every random generator of theories C G AA{Q), v-almost 
surely, R^{6) is the complement of a first-category set of paths. 

Assume that Alice tests Bob with a randomized likelihood test and that 
Bob uses an arbitrary random generator of theories C. Proposition 3 shows 
that, with {i-probability one, Alice selects a prequential test T^J^ for which 
there exist a topologically large set of paths that, if realized, ^-almost surely, 
reject Bob's theory. 

We now provide a general result showing that random prequential tests, 
including the randomized likelihood test, are manipulable as far as a natural 
generalization (to random tests) of definition 3 of manipulability goes. 

Proposition 4. Fix any 5 > 0. Let (@,B,v) and f be a prequential 
random test (satisfying the joint measurability condition) that does not reject 
the data- generating process with probability 1. Then, there exists a random 
generator of theories Q such that, on any path s G 0, 

C({P G A(0) : v-almost surely f{e, s, P) = 1}) > 1 - 5. 

The random generator of theories ^ may depend upon 6, {Q,B,v) and T, 
but it does not require (for its construction) any distributional assumptions 
over the future realizations of the paths. Proposition 4 shows that Bob can 
produce theories according to a random device ^ such that, no matter which 
path s is realized, it is unlikely (odds given by v and (") that Alice selects a 
test that rejects Bob's theory. 

To reconcile Propositions 3 and 4, consider the randomized likelihood 
test (0,9,'u), T^^. This test satisfies the joint measurability conditions, 
and so, the conclusions of Propositions 3 and 4 hold for this test. Let E'" 
and be the expectation operators associated with v and (, respectively. 



A NONMANIPULABLE TEST 



15 



By Proposition 3, for every C € AA(r2), 




[sec 



By Proposition 4, for every 5 > there exists C, G AA(ri) such that 



From (4.3), Bob knows that Ahce (almost surely) selects a test for which 
there are paths that, if realized, will (almost surely) reject Bob's randomly 
selected theories. From (4.4], Bob knows that if he selects a theory with 
carefully designed odds (by Q), then (no matter how the data evolves in the 
future) it is unlikely that the selected theory will be rejected by Alice's test. 

Consider the question of whether a strategic, but uninformed, expert can 
pass a random prequential test (which does not reject the data- generating 
process). Proposition 3 seems to answers this question in the negative while 
Proposition 4 seems to answer this question in the positive. Hence, results 
(4.3) and (4.4) leave room for different interpretations. In our viewpoint, 
Proposition 4 is a natural generalization (to random tests) of existing re- 
sults showing that prequential tests are manipulable. Our preference (for 
result 4 over 3) can be understood in the context of decision-making under 
uncertainty as described in Section 3.1. 

Assume that Alice offers a contract to Bob. If Bob accepts the contract, 
he delivers a theory to Alice and receives positive payment. However, if 
Bob's theory is rejected by the test selected by the randomized likelihood 
test, then Bob is penalized. Now, assume that Bob knows nothing about 
the data-generating process. Then, Bob faces uncertainty about the proba- 
bilities of the future realizations of the data. By definition. Bob knows the 
odds that Alice uses to select her test, and he also knows the odds that 
he uses to select his theory. Hence, with regards to theory selection and 
to test selection. Bob faces common risk. The most pessimistic behavioral 
rule of decision under uncertainty, among those axiomatized by Gilboa and 
Schmeidler (1989), determines Bob's prospects by his expected utility com- 
puted in the worse-case scenario [as in (4.4)]. By this rule of decision under 
uncertainty, Bob accepts Alice's contract. Hence, prequential random tests 
do not screen informed and uniformed experts. 

The results showing that prequential tests (and even randomizations over 
prequential tests. Proposition 3 withstanding) are manipulable, combined 
with the fact that some nonprequential tests are nonmanipulable, poses a 
difficulty for the prequential principle (as far as testing potentially strate- 
gic experts goes). However, this difficulty may not persist under conditions 
that are beyond the scope of this paper. Fortnow and Vohra (2007) show a 
prequential test that is computationally demanding to manipulate. In addi- 
tion, Olszewski and Sandroni (2008) show a prequential test that cannot be 



(4.4) 



inf E^E^{T^^} > 1 - (5. 
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manipulated when the domain of permissible theories (i.e., the theories the 
expert is allowed to announce) is restricted. It is not known whether the re- 
sults in this paper extend to the case of multiple experts [see Al-Najjar and 
Weinstein (2007) and Feinberg and Stewart (2007) for some results on test- 
ing several experts simultaneously]. Finally, while any given prequential test 
(which does not reject the data-generating process) can be manipulated, it is 
not possible to manipulate all prequential tests (that do not reject the data- 
generating process) simultaneously [see Olszewski and Sandroni (2008)]. 

4.3. Nonprequential manipulable tests. As mentioned in the Introduction, 
prequential tests are manipulable. So, prequentiality is a sufficient condition 
for manipulability, but it is not a necessary condition. We now show several 
tests, many of them nonprequential, that can be manipulated. 

Definition 7. Acceptance tests have the property that, for any theory 
P £ A(il), the acceptance set Ap is a union of cylinders. 

In an acceptance test, the acceptance sets are open. In a rejection test, 
the rejection sets are open. 

Proposition 5. Fix any e G [0, 1] and 5 € (0, 1 — e]. Let T be an accep- 
tance test that does not reject the data- generating process with probability 
1 — £. Then, the test T can be manipulated with probability 1 — e — 5. 

A formal proof of Proposition 5 is presented in the Appendix. An intuition 
is as fohows: let V : A{n) x AA{n) [0, 1] be a function defined by V{P, C) = 
E^E^T; that is, V{P, C) is the probability of the verdict 1 if P is the data- 
generating process and C is the random generator of theories used by Bob. By 
assumption, for every P G A(Q), there exists S AA(r2) (a deterministic 
generator of theories that assigns probability one to P) such that V{P, (p) = 
1 — £. Thus, if the conditions of Fan's minmax theorem are satisfied, then 
there also exists Ct G AA(il) such that V{P, Ct) > 1 — e — 5 for every P G 
A(f]). This yields the result, since V{P,Ct) = CtHQ G A(0) | T{s,Q) = 1}) 
if P is the degenerated measure that assigns probability one to s. 

As is well known, A(0) is compact in the weak*-topology and y is a 
bilinear function. Hence, the conditions of Fan's minmax theorem (see the 
Appendix for this result) are satisfied if V is lower semi-continuous with re- 
spect to P. We show that this lower semi-continuity follows from the open- 
ness of acceptance sets. It is here that the assumption that T is open turns 
out to be essential. In the case of a rejection test, V is upper semi-continuous 
with respect to P, but not necessarily lower semi-continuous. 
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APPENDIX: PROOFS 

Proof of Theorem 1. Define Al{P) :=Q- C {s\f^- p-^) as the com- 
plement of (^(^^(j p))- Let 

oo oo oo 

AkiP):=f]AiiP) and Ap := \J A^iP) U \J {s'} . 

i=l k=l 1=1 

Each set Ak{P) is an intersection of closed sets and is therefore closed 
itself. By construction, each set Ak{P) has an empty interior; indeed, the 
complement of A^^^P); that is, the set 

CXD 

o-^,(p) = Uc(sj(,fc,P)) 

j=l 

is open and dense. Hence, Ap is a first-category set. Notice that Ap is the 
acceptance set of P G A(r2). 
Since 



- ip C - U U {s'} ] C U [C{s\^i,k,P)) - {s'}] 

for all k£N, 



i=i / i=i 



p{n - ip) < E p(c{si(^^,k,P)) - is'}) < E 



oo 2^ 



2k+i 2^ ' 
i=l i=l 

also for all k ^ N, which yields that P{Ap) = 1. Thus, the global category 
test does not reject the data-generating process with probability one. It 
remains to show that the test cannot be manipulated. Suppose we are given 
a € AA(r2). We first show that there exists a subset A 0,, which is a 
countable union of closed sets with empty interior, and a Borel set B C A(0) 
such that 

(A.l) CiB) = l 

and 

(A.2) Vpes ApcA. 

We show later (in Corollary A.l, which follows this proof) that for every 
s the set {P e A{Q) : s e Ap} is Borel. Since 

B C{P £A{n):s^Ap} 

for every s & Q — A, we obtain that 

y^en-A C({^GA(O):sGip})=0. 
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This means that ^l — A C R'^, and so the complement of is a first category 
set. 

We will now construct sets A and B with properties (A.1)-(A.2). Consider 
the sets 

Bl{m) := {P G A(0) : t{i, k, P) > m}; 

Lemma A.l, which follows this proof, shows that the sets B\[m) are open 
and so Borel. Since this sequence of sets is descending (with respect to m, 
for any given k and i) and its intersection is empty, for every Z = 1,2,... 
there exists an m such that 

denote by m^(Z) any such m. 
Let now 

4(0:=f^-C(sj„) form = ml(/), 

oo 

A,{l):={^Am 

i=l 

and 

oo oo 

Ail) :=[jAk{l)u[j{s^}. 

k=l i=l 

The set A(l) is a countable union of closed sets with empty interior by an 
argument analogous to that used, in the main body of the paper, for the 
case of the sets Ap, and so is 

oo 

A:=[jAil). 

1=1 

To show that (A.l) and (A. 2) are satisfied, notice that, by the definition of 
B\{m) for m = m\{l) , \i P ^ B\{m), then C(s^) C C{s\^^ k P))'' therefore, 

oo 

if P G A{n) - y Biimlil)) then ^.(P) C A^il), 
1=1 

which in turn yields that 

oo oo 

if P G A{n) - U U Biimiil)) then Ap C A{1). 

k=li=l 

Thus, 



B:=[j 

1=1 



A{n) - [j [j BUmlil)) 

k=li=l 



C{P£A{n):ApCA}. 



A NONMANIPULABLE TEST 19 
It remains to show that C{B) = 1; however, 



oo oo 



aB)>C[A{n) - [j [j BlimW))] 

\ k=li=l ) 

oo oo 

>i-EEc(^i^«(0)) 
fc=i j=i 

oo oo 2 2 

- ''^ ~ E E nk+i+l Ql 
k=li=l 

for every I G N. □ 

Lemma A.l. For every t G A^, the set 

Bl{t) :={PGA{n):t{i,k,P)>t} 

is open. 

Proof. Let P G B\{t). By definition, t{i,k,P) > t, which means that 

1 



PiC{sl)-{s^})> 



2k+i 

Further, there exists an m > t such that 



(A.3) P(C(sj)-C(4))> ^ 



2k+i ' 

indeed, the sequence of sets C(sJ) — C(s^) is ascending (as m goes to infin- 
ity), and its union is equal to C{sl) — {s*}. 

Note that each cyhnder is an open and closed subset of and so is 
C{sl) — C{sln). Thus, the function f -.Q^ R given by 



fis) 



1, for every s G C(sJ) — C(s* 

'i) ~ C!{sl^j, 



0, for every s ^ C(sJ) - C(s* 



is continuous. 
Let 



(A.4) S:=P{Cisi)-C{sl))-^, 

and let N{P) stand for the set all measures Q G A(0) such that 

JfdQ-J fdP 
This last inequality means that 

\Q{C{sl) - C{sl,)) - P{C{s\) - C(4))| < 5; 
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by (A.3) and (A.4), 

QiCisl) - {s^}) > Q{C{s\) - C{s^J) > l/2'=+\ 

which imphes that Q G (t) . 

That is, the set N{P), which is an open neighborhood of P in weak*- 
topology, is contained in Bl{t). □ 

Corollary A.l. For every sgQ., the set {P G A($7) : s G Ap} is Borel. 

Proof. If s = for some i G N, then {P G A(0) : s G Ap} = A(J7) by 
definition. Suppose, therefore, that s ^ s^ for any i ^ N . It suffices to show 
that the sets {P G A(i7) : s G C{s\^^ fc p))} ^■'^^ Borel, as 



{PG A(!^):sGip} = A(J^) 



n U{^eA(J7):sGC7(si(,,,p))} 

Lfc=l \i=l / 



Since s ^ s^ and Cp(so) = A(r2), there is a unique m = 0, 1, . . . such that 
s G C^(s^,) - C7|,(sj„+i). For this m, we have that {P G A(J1) : s G C(4(._fc p))} 

A(J^) - {P G A{n) : t{i, k, P)>m}, and so the set {P G A{n) : s G C(si(,,fc,p))} 
is closed by Lemma A.l. □ 



Proof of Proposition 2. First, consider P g [Ay. It follows from 

1 

(m+l)^ 



(4.2) and from P(P^) < that, for any cyhnder C C P™ 



By definition, Qp{A) > 7r(m)Q^(^) = (^+i)^ Qp(^) for any m > 1 and 
set ^ G 9. Hence, for any cylinder C C i?p , 

(A.5) Qp{C) > ("^ + y p(c) > ^p(c). 

Given a theory P G (^)'^, let i2p and Rp be the rejection sets of P for the 
tests Ts and T, respectively. Consider a path s G i?p. Then, s G i?p for every 
m > 1, because T"* is harder than Ts- Since Pp comprises cylinders, there 
is a cylinder C{sn{m)) with base on = s \ n{m) such that C{sn{m)) ^ 

R'pcRp. By (A.5), 

<3p(C(Sn(m))) . ,. Qp(C('Sn(m))) 
— ■ ^ TTi y iim — = (X) 

P{C{Snim))) - P{C{Snim))) 

Thus, s G Rp. So, if P G {Af, then Pp C Pp. 
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Now, consider a theory P A. Consider a path s £ Rp. Given that P 
assigns probabihty one to the union of all its atoms, it follows that s is 
not an atom of P. So, P{C{st)), st = s \t, approaches zero as t goes to 
infinity. Since P has finitely many atoms, there exists an r/ > such that 
the probability of each atom is smaller than rj. Let t be large enough so that 
P{C{st)) < T]. It follows that C(sj) contains no atom. Hence, P{C{st)) = 0. 
So, s & Rp. It follows that T is harder than T. □ 

Remark A.l. The existence of a cylinder with finite base that satisfies 
(4.1) can be shown as follows. 

If P G (Ay, then there are infinitely many atoms, or the set of all nonatoms 
has positive measure. If there are infinitely many atoms, then one of them 
(say s) has to have measure below ^^^j^^^p . It follows that the P{C{st)), 
St = s \t, must be strictly positive and (for t large enough) smaller than 
2(m+i)^ ■ Now, consider the case in which the set of all nonatoms of P has 
positive measure. Assume that every nonatom s S of P is contained in a 
cylinder Cs (with finite base) such that P assigns zero probability to Cg ■ Let 
C be the union of all zero-probability cylinders. There are only countably 
many cylinders with finite base. So, P{C) = 0. Given that C contains all 
nonatoms of P, it follows that the set of nonatoms has zero measure. This 
is a contradiction. So, P has a nonatom s such that P(C(sj)) 7^ 0, st = s \ t, 
for all t £ N. Since P{C{st)) approaches zero as t goes to infinity, P{C{st)) 
must (for t large enough) be smaller than 2(^+1)^ • 

Lemma A. 2. The test satisfies the joint measurability condition. 
Proof. We need show that the set 



{{9,s,P)£Gxnx A{n) : r^"(0, s, P) = 1} 



is measurable. Notice that 



{{0 



s,P) ee X 17 X A{n):T^^{6, 



s,P) = l} 




V„gAfP(C'('Sn)) 7^0 and lim sup 



Qs,{C{sn)) 

PiC{Sn)) 



< 00 



} 



e X X A(0) 



00 00 



fl U {{9, s, P) G e X X A(0) : Qss{C{sn)) > A;P(C(s„))} 



fc=ln=l 
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where (as always) Sn = s \ n. To see the last equality, notice that Q5g(C(s„)) > 
for every cylinder C{sn), so if P(C(s„)) = 0, then the inequality Qsg{C{sn)) > 
kP[C{sn)) is satisfied. Therefore, it suffices to show that sets of the form 

G e X J] X ^{n):Qs,{C{sn)) > kP{C{Sn))} 

are measurable. 

Further, the inequality that defines this last set depends only on s„, not 
on the entire path s, so this set can be represented as a union of sets 

{{6, P) e e X A{n) : Qso{C{Sn)) > kPiCiSn))} X C{Sn), 

and so it suffices to show that sets of the form {{6, P) G x A{Q,) : Qsg (C) > 
kP{C)}, where C is a given cylinder (with base on s„), are measurable. We 
will show that every set of this form is open. 

Indeed, a pair {9,P) belonging to this set means that 

denote by 77 > the difference between the two expressions. Take the set of 
all paths Q such that 

(A.7) E ^>M:=.P(C) + |, 



and the set of all probability distributions P such that 

\P[C) -P{C)\ < ^. 

Obviously, the Cartesian product of the two sets contains the pair {6,P), 
and any pair {6, P) that belongs to this Cartesian product satisfies condition 
(A. 6) (for P replaced with P and 6 replaced with 6). So, it suffices to show 
that the two sets are open. The fact that the latter set is open follows directly 
from the definition of weak*-topology, as any cylinder C is a closed and open 
subset of Q. 

We will now show that the former set is also open. If 9 satisfies (A.7), 
then 

- 1(1 + 1) 

for some t = 1, 2, . . . , where 

CX9) := Cn {0, 1}* x {/t+i(^)} X {/t+2(^)} X . . . . 

This follows from the assumption that the paths from {0,1}* x {/i(0)} x 
{/t+i(6')} X • • • precede the paths from Se - {0, 1}* x {/i+i(0)} x {/t+2(6')} x 
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•••in the sequence Sq,Sq,... and the fact that, if an infinite sum exceeds 
some number, then sufficiently large finite sums exceed that number as well. 

With no loss of generality, one can assume that t>n; recall that n is the 
length of the base of C. Then, for any 9 e C{6t), the cylinder with base on 
the first t outcomes of ^, we have 

y y ^-^= y ^^-^>m; 

? e 

the middle equality follows from the fact that the first t outcomes of 9 and 
9' coincide, and so s'(e) G C\9) is equivalent to s'(e') £ C\9'). □ 

Lemma A. 3. Let S = {s^, s^, . . .} be a countable dense subset ofH,. Take 
any P € A(il) such that P{S) = 0. The set of paths on which Tg^ rejects P 
is a superset of the set of paths on which the global category test Ts rejects 
P. 

Proof. Take any s efl such that fs{s, P) =0 and /c = 1, 2, Then, 

s £ C{sl^-f^ps^) for some i = l,2,...; in other words, St(i^k,P) = ^l^^j^py On 
one hand, as P{S) = 0, 

and, on the other, 



i{i + l) 

Thus, 

P{C{st^i^k,P))) - + - 
for every = 1, 2, . . . , which means that the sequence 



M£i))- 



p(C(st)) h=i 
bounded. □ 



Proof of Proposition 3. We show that, except countably many 
paths 9, the set R^{9) is a superset of the revelation set of the global 
category test Ts for S = Sq. Suppose that R^{9) is not a superset of the 

revelation set R^ of the global category test Ts for S = Sg. Then, by Lemma 
A. 3, it must be the case that 



C{{PeA{n):P{Se)>0})>0. 
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If this is the case for an uncountable number of G 0, then there exist 
m,n N such that 

c({p€A(n):P(s„)>i})>i, 

for an uncountable set H C 0. 

Notice that the set Q can be partitioned into countable subsets such that 
9^ and 9'^ belong to the same subset if Sqi = Sq2] that is, paths 9^ and 9"^ 
coincide on all but a finite number of outcomes. Notice further that, if 9^ 
and 9^ belong to the distinct subsets, then the sets Sqi and 5^2 are disjoint. 
Without loss of generality, we can assume that distinct paths from H belong 
to distinct subsets; that is, they differ on an infinite number of outcomes. 
Therefore, for any P G A(0), P{Sq) > 1/m for at most m — 1 paths G H. 
Further, we can restrict attention to an infinite but countable subset of S; 
from now on, we will denote this subset by H. 

Consider any linear ordering ^ of the set H, and define sets 

:= \P G A(n) : 9 is kth path from E such that P(Se) > — \ 
[_ m } 

for k = 1, . . . ,m — 1 and G H. 

For a given k = 1, . . . ,m — 1, the sets Dg are pairwise disjoint, and, for 
a given G H, the sets Dq are pairwise disjoint. The measurability of the 
sets Dg follows from the measurability of the set of all measures P that at a 
given path s have an atom of measure larger than a given number. Finally, 



(p G Ain) : P{Se) >-]=Dlu---U D^'-^ 



for every G H. 
Thus, 



m - 1 = {m - 1) ■ CiA{n)) 

m— 1 / \ m— 1 / 

k=i \0es / k=i \ees 

ees\fc=i / 
= J2c({PeA{n):P{Se)>- 



a contradiction. □ 
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Proof of Proposition 4. Let T' :Qx A(J7) {0, 1} be a test defined 
by f'{s, P) = 1 if and only if f{9, s,P) = l u-almost surely. The test f' is a 
prequential test because, given two theories P and P' equivalent along s, for 
every ^ G G, f s, P) = 1 if arid only if f{e, s,P') = l. Hence, f{e, s,P) = l, 
{j-almost surely, if and only if T{6, s, P') = 1, -y-almost surely. Since, for every 
6'Ge, 

p({sgo |r(0,s,p) = i}) = 1, 

by Fubini's theorem 

P{{s G I f{e, s,P) = l v-almost surely}) = 1. 

So, T" does not reject the data-generating process with probability one. 
By Shmaya's (2008) result [which relies on Martin's (1998) theorem], there 
exists Q such that for all s G 

C({PGA(f^)|r'(s,P) = l})>l-5. □ 

Let X be a metric space. Recall that a function f '.X ^ R is lower semi- 
continuous at an X G X if, for every sequence {xn)^=i converging to x, 

V£>0 3n V„>Ar f{Xn) > f{x) - E. 

The function / is lower semi-continuous if it is lower semi-continuous at 
every x gX. We refer the reader to Engelking [(1989), Problem 1.7.14] for 
these definitions and some basic results regarding lower semi-continuous 
functions. 

Lemma A. 4. Let U C X be an open set where X is a compact metric 
space. Equip X with the a-algehra of Borel subsets. Let A{X) be the set of 
all probability measures on X. Equip A(X) with the weak* -topology. The 
function F : A(X) [0,1] defined by 

F{P) = P{U) 

is lower semi- continuous. 

Proof. See Dudley (1989), Theorem 11.1.1(b). □ 

Theorem [Fan (1953)]. Let X be a compact Hausdorff space, which is 
a convex subset of a linear space, and let Ybe a convex subset of linear space 
(not necessarily topologized) . Let f be a real-valued function on X xY such 
that for every y €Y, f{x,y) is lower semi- continuous, with respect to x. If 
f is also convex, with respect to x, and concave, with respect to y, then 

minsup/(x,y) = supmiri/(3;,y). 
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We note that Fan's (1953) theorem allows for X and Y that may not be 
subsets of linear spaces. We, however, apply his result only to subsets of 
linear spaces. 

Proof of Proposition 5. Let X = A(r2), let Y be the subset of 
A(A(il)) that consists of all random generators of theories with finite sup- 
port. So, an element C of y can be described by a finite sequence of proba- 
bility measures {Pi, . . . ,-Pn} and positive weights {vri, . . . ,7r„} that add up 
to one, where C, selects Pi with probability vTj, i = 1, . . . ,n. Let the function 
/ : X X y ^ P be defined by 

n „ 

(A.8) f{P,0:=EPE^T = Y^TT^ T{s,P)dP{s). 

i=l 

We now check that the assumptions of Fan's theorem are satisfied. Since 
T is an open test, the set 

Uq = {s(^VI:T{s,Q) = 1} 

is open for every Q G A(r2). Therefore, by Lemma A. 4, 

P{Uq) = J T{s,Q)dP{s) 

is a lower semi-continuous function of P. Thus, for every ( £Y, the function 
f{PX) is lower semi-continuous on X as a weighted average of lower semi- 
continuous functions. 

By definition, / is linear with respect to both x and y, and so it is convex 
with respect to x and concave with respect to y. By the Riesz and Banach- 
Alaoglu theorems, X is a compact space in weak*-topology; it is a metric 
space, and so Hausdorff [see, e.g., Rudin (1973), Theorem 3.17]. 

Thus, by Theorem 1, 

min swpE^ E'^T = sup min E^E^'T. 

Notice that the left-hand side of this equality exceeds 1 — e, as the test T 
is assumed not to reject the data- generating process with probability 1 — e; 
indeed, for a given P £ X, take C such that C({-f }) = 1- Therefore, the right- 
hand side exceeds 1 — £, which yields the existence of a random generator 
of theories C ^ ^ such that 

E^E'^T >l-e-5 

for every P G A(0). Taking, for any given s E $7, the probability measure P 
such that P({s}) = 1, we obtain 



C({Q G A{n) : r(s, Q) = l})>l-e-5. 



□ 
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